feat(storage): add object contexts in Python GCS SDK#17039
Conversation
Implement object contexts feature parity with Go and Java SDKs. Allows users to attach custom key-value metadata payloads to objects and filter by them during list operations. - Define ObjectCustomContextPayload and ObjectContexts classes in blob.py. - Add 'contexts' property to Blob class and include it in _WRITABLE_FIELDS. - Update list_blobs in Client and Bucket to support 'filter_' parameter. - Implement gRPC conversion logic and update_mask generation in _grpc_conversions.py. - Add comprehensive unit tests. Co-authored-by: nidhiii-27 <224584462+nidhiii-27@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for object contexts in the Google Cloud Storage Python SDK, adding the ObjectContexts and ObjectCustomContextPayload classes and a contexts property to the Blob class. It also implements server-side filtering for list_blobs via a new filter_ parameter. Feedback focuses on improving the robustness of protobuf conversions by checking for field presence rather than truthiness, ensuring FieldMask generation for contexts replaces the entire map for consistency with other SDK fields, and hardening the custom context setter to handle None values.
- Improve gRPC conversion logic: use 'in' operator for field presence check and improve property mapping. - Refine update_mask generation logic. - Update tests to match new conversion logic. Co-authored-by: nidhiii-27 <224584462+nidhiii-27@users.noreply.github.com>
- Implement ObjectContexts and ObjectCustomContextPayload. - Add 'contexts' property to Blob. - Add 'filter_' parameter to list_blobs. - Add gRPC conversion for contexts and update_mask generation. - Fix unit tests to avoid ADC errors. - Format code with ruff. Co-authored-by: nidhiii-27 <224584462+nidhiii-27@users.noreply.github.com>
| custom_contexts = {} | ||
| for key, payload in contexts.custom.items(): | ||
| payload_params = {"value": payload.value} | ||
| if payload.create_time is not None: |
There was a problem hiding this comment.
can we set create_time and update_time ?
looks like they are output only - http://shortn/_c796zOs1VO
There was a problem hiding this comment.
@jules remove the create_time and update_time
There was a problem hiding this comment.
Removed create_time and update_time from blob_to_proto as requested.
There was a problem hiding this comment.
this file will be used grpc ,hence can you write one system test for zonal buckets ?
There was a problem hiding this comment.
@jules add a system test in test_zonal.py
There was a problem hiding this comment.
Added a system test for object contexts in test_zonal.py.
There was a problem hiding this comment.
Please add system test for
- Adding object context while uploading object
- list blobs with object context filter
Refer - https://docs.cloud.google.com/storage/docs/use-object-contexts#filter_objects_by_contexts
There was a problem hiding this comment.
Added the required system tests for object contexts and list filtering in test_blob.py.
There was a problem hiding this comment.
provide option to modify object contexts during
rewrite, copy and compose - https://docs.cloud.google.com/storage/docs/use-object-contexts#manage_object_contexts_during_object_operations
There was a problem hiding this comment.
@jules add this option to the methods above
| @@ -1400,6 +1401,9 @@ def list_blobs( | |||
| Note ``soft_deleted`` and ``versions`` cannot be set to True simultaneously. See: | |||
| https://cloud.google.com/storage/docs/soft-delete | |||
|
|
|||
| filter_ (str): | |||
There was a problem hiding this comment.
add link to the syntax for filter - https://docs.cloud.google.com/storage/docs/listing-objects#filter-by-object-contexts-syntax
There was a problem hiding this comment.
@jules modify the docblock and add the link
| @@ -1516,6 +1517,10 @@ def list_blobs( | |||
| Note ``soft_deleted`` and ``versions`` cannot be set to True simultaneously. See: | |||
| https://cloud.google.com/storage/docs/soft-delete | |||
|
|
|||
| :type filter_: str | |||
There was a problem hiding this comment.
add link to the syntax for filter - https://docs.cloud.google.com/storage/docs/listing-objects#filter-by-object-contexts-syntax
There was a problem hiding this comment.
@jules modify the docblock and add the link
- Remove create_time and update_time from gRPC conversion. - Add system tests for Object Contexts in test_blob.py and test_zonal.py. - Format code with ruff. Co-authored-by: nidhiii-27 <224584462+nidhiii-27@users.noreply.github.com>
2 participants
This PR implements the "Object Contexts" feature in the
google-cloud-storagePython SDK, ensuring feature parity with the Go and Java SDKs.Key changes:
ObjectCustomContextPayloadandObjectContextsclasses togoogle/cloud/storage/blob.py. These allow users to define custom key-value pairs (contexts) for objects.contextsproperty to theBlobclass with appropriate getters and setters. Addedcontextsto_WRITABLE_FIELDSto enable REST API support for patching and updating these contexts.list_blobsmethod in bothgoogle/cloud/storage/client.pyandgoogle/cloud/storage/bucket.pyto include a keyword-onlyfilter_parameter. This allows server-side filtering of objects based on various attributes, including custom contexts.google/cloud/storage/_grpc_conversions.pywith:blob_to_proto: Support for convertingcontextsto GCS V2 proto.proto_to_blob: Support for populatingBlob.contextsfrom GCS V2 proto.get_update_mask: Logic for generating gRPCFieldMaskfor partial context updates (using shallow masks likecontexts.custom.<key>).google/cloud/storage/tests/unit/test_blob.pyandgoogle/cloud/storage/tests/unit/test__grpc_conversions.pyto verify the new functionality and ensure no regressions (verified withnox -s unit-3.12). Fixed a minor regression in async write tests.This implementation allows for advanced metadata management and powerful server-side filtering as requested.
PR created automatically by Jules for task 13325527155543531515 started by @nidhiii-27